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(57) Abstract: The invention relates to a system 
for operating with different types of media content, 
the system being arranged to enable a user to use 
a first content of a first type. The system comprises 
identifying means (120) for identifying that the 
user concurrently uses a second content of a second 
type, said second content being unrelated with 
the first content, and associating means (210) 
for associating said second content with the first 
content The invention also relates to the method of 
operating with the different types of media content 
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System and method for associating different types of media content 



The invention relates to a system for operating with different types of media 
content, the system being arranged to enable a user to use a first content of a first type. 

The invention also relates to a method of operatmg with different types of 
media content, the method comprising a step of identifying a user's usage of a first content of 
5 a first type. 



Document US2002/0002564A1 discloses data processing means for viewing, 
playback and deUvery of continuous audiovisual data, media content such as motion pictures, 

10 video programs, audio programs, etc. The media content, e.g. MPEG-1 system stream or 
other format, and description data describing audio and video content are received by said 
means firom outside, or said content and data can be stored in a database firom which they can 
be retrieved. The description data are expressed through the use of, e.g. Extensible Markup 
Language (XML) whose standardization is pursued by the World Wide Web Consortium. 

15 The description data describe parameters of the audio and video content such as a title, 
caption, priority, location, start time, end time, etc. Video and audio streams of the media 
content are mutually synchronized and provided to video playback means and audio playback 
means. The media content is fiirfher rendered for the user. In the known data processing 
means, the user may request or disable simultaneous rendering of the provided audio and 

20 video contents. The user may choose to be provided with only the audio part or only the 
video part of the available media content. 

In the known data processing means, the audio and video contents correspond 
to each other because they are received together firom a content provider, e.g. a TV 
broadcaster or the like. At the same time, the user may like simultaneous rendering of media 

25 contents which are not related by the content provider. It is a drawback of the prior art that 
user's preferences with respect to rendering the different contents are not taken into account. 
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It is an object of the present invention to provide a system for operating with 
different types of media content of flie kmd defined in the opening paragraph which does take 
into accoimt the user's preferences with respect to simultaneous rendering of the diflEerent 
contents. 

The object of the invention is realized in that the system comprises identifying 
means for identifying that the user concurrently uses a second content of a second type, said 
second content being unrelated with the first content, and associating means for associating 
said second content with the fiist content. 

The contents of different types, e.g. audio and video, which are not associated 
by any reference in any way in the system may be related in the user's mind. For example, 
the user may like simultaneous rendering of some contents of the different types, or the 
combination of the contents may carry the usct back to some life experience, or the like. The 
system comprises identifying means and associating means for •'registering'' the relationship 
between such contents in the system. The identifying means identifies that the user uses, e.g. 
watches, hstens to, etc, a first content of a first type and, possibly in a different way, a second 
content of a second type at the same time, whereia said first and second contents are not 
related in the system. The associatiag means associates said first and second contents to 
enable the system to exploit this relationship. 

One of the advantages of the present invention is that the user does not have to 
explicitly instruct the system to relate such contents because it may be done automatically 
and transparently for the user without hindering the user's usage of said contents. Another 
advantage is that the links between contents are made personal to the user of the system, 
which is not possible when the contents, e.g. a TV program or a movie which usually include 
audio and video, are obtained from the content provider. 

The object of the present invention is also realized in that the mefliod of 
operating with different types of media content further comprises a step of identifying that 
the ixser concurrently uses a second content of a second type, said second content being 
imrelated with the first content, and a step of associating said second content with the first 
content. The method describes the functioning of the system of the present inventioiL 

These and otiier aspects of the invention will be further elucidated and 
described with reference to the accompanying drawings, wherein: 

Fig. 1 illustrates the general scheme of the present invention; 



wo 2004/029835 PCT/IB2003/003660 

Fig. 2 shows a functional block diagram of the system suitable for 
implementing the present invention; 

Fig. 3 shows an example of the associated contents of different types and the 
relationship between them; 

Fig, 4 shows an example of a time sequence of the user's usage of different 
media contents; 

Fig. 5 shows an embodiment of the method of the present invention. 
Throughout the Figures, the same reference mmierals indicate identical or 
corresponding components. 



Many people today use a lot of media content. People watch, listen, browse 
different audio and video information, as well as create their own media content, e.g. by 
voice recording, taking pictures using a camera, making a home film, etc. Various consxmier 
electronic devices 110 such as a television set (TV set) with a cable, satellite or other link, 
video cassette recorder (VCR), home stereo, internet broadcasts, world wide web, portable 
and stationary radio, CD player, personal digital assistant (PDA), etc. can be used for these 
purposes, as is shown in Fig. 1 . Each of these devices may have a user interface used to 
obtain, select, record, play, etc. the media content. Any one of the consumer electronic 
devices allows the user to use at least one media content, and some of these devices may 
output or input at least two contents of different types. The system according to the present 
invoation comprises identifying means 120 for idoatifying tibiat the user concurr^itly uses at 
least two contents of different types, whereas these contents are not imrelated in the system, 
e.g. the system does not include data indicating the relation between said contents. Such 
identifying means 120 may be any means capable of monitoring user's activities in the 
system, e.g. a monitoring camera connected to a dedicated data processing imit which is 
arranged to analyze the user's behavior by processing data obtained from the camera, said 
data processing unit may be a part of the consmner electronic device rendering the media 
content. 

One or many media devices capable of outputting and/or inputting the media 
content may be included in the system. When a plurahty of such devices is included, they 
may have means (not shown) for cormnunicating with each other, e.g, the devices may be 
connected into a network. The communication means may be arranged to obtain information 
about the user's usage of tiie content from each of the devices, and recognizing individual 
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devices providing said information. For example, each device outputs to the communication 
means its identification and information pertaining to the contents used by the user, e.g. the 
type of the used content(s) which may be one of predetermined types in the system, etc. 
Further details which concern functioning of the identifying means will be apparent to the 

5 person skilled in the art. 

The media content may basically comprise at least one or any combination of 
visual information, audio inforaiation, text, data describmg the media content or its 
reproduction, or the like. The expression audio data, or audio content, is hereinafter used as 
data pertaining to audio comprising audible tones, silence, speech, music, tranquility, extemal 

10 noise or the like. The expression video data, or video content, is used as data which are 
visible such as a motion picture, static image, characters etc. The user device may be 
arranged to obtain the media content stored on different carriers such as, audio tapes, video 
tapes, optical storage discs, floppy and hard drive disks, etc. in any format, e.g. MPEG 
(MoviQg Picture Experts Group), MIDI (Musical Instrument Digital Interface), Shockwave, 

15 QuickTime, WAV (Waveform Audio), etc. The content may also be broadcast by a content 
provider, e.g. TV or radio, to be received by the user devices rendering said content for the 
user. The user may use the media content at home, pubUc places, in public transport and so 
on. Examples of the different types of content may be a playlist of songs or music, moving 
pictures, still images or sets of the still images, e.g. an image slide, animation, text, meta-data 

20 related to the media content, etc. For instance, the meta-data may define the radio content, 
e.g. radio stations and the most frequently broadcast content that is available in the user's 
liviQg area, the different kinds of playlists for setting up a collection of the media contents. 
The meta-data may, for example, include the music playlist, e.g. in which the next song is 
started when the curr^t one ends, the radio station or television chaimel playlist, e.g. 

25 comprising a collection of favorite channels to watch or surf through. 

With reference to Fig. 2, a functional block diagram of the system of the 
present invention is shown. The system 200 comprises the identifying means 120 and 
associating means 210 for associating said two or more contents, simultaneous use of which 
is identified by the identifying means 120. The identifying means outputs to associating 

30 means tiie information concerning the contents concurrently used in the system. The 

associating means may be arranged to establish data which link said associated contents, such 
as meta-data comprising Ihe information pertaining to a relationship between the associated 
contents. In one example, the meta-data may include information about the associated 
contents such as a name/title, source of or link to the contents, e.g. identification of the 
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device(s) used for rendering the content; observed user's behavior during the use of the 
content(s), e.g. types of the user's behavior like watching, listening, etc. may be 
predetermined, whether or not the user initiated the content rendering, date, start and/or ond 
time of the content usage, duration of rendering, etc. as is shown in Table 1. Of course, some 
5 of the information included in the meta-data may be obtained from the identifying means 
120. 



TABLE 1 
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10 The associating means may be arranged to associate the contents automatically 

or to communicate information pertaining to the contents to the user for obtaining his/her 
approval for associating these contents. The system may also simply notify the user about 
associating the corresponding contents. 
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The system may optionally comprise storage means 220 arranged to store the 
meta-data, selection means 230 arranged to select the content, output means 240 arranged to 
output the associated contents, and rating means 250 arranged to rate the associated content. 

The meta-data may be implemented in any standard data structure or database. 



contents which is sufficient to distinguish them. In one example, the meta-data may be stored 
in the storage means 220 together with the corresponding contents, e.g. in a header of the 
corresponding media data. The new meta-data may be established whenever some contents 
are associated by the associating means. Thus, a plurality of meta-data records may be stored 
by the storage means. 



selection means 230 may be arranged to identify the first content upon selection of the 
associated second content, and vice versa. Thus, the system can always find all associated 
contents if one of them is selected and the relationship between these associated contents has 
been estabhshed. The ou^ut means may be arranged to play, reproduce, etc. the associated 
contents simultaneously. 



the media content. The recommender may exploit the meta-data for recommending the 
content in accordance with the user's prefermces. In that way, the recommendations can be 
personalized for the user. 

The recommender may be arranged to recommend the associated contCTts if 
the user selects one content and likes to be reconmiended with another content of different 
type in accordance with his/her preferences. The recommender may be further arranged to 
recommend contents correlating to one of the associated contents e.g. having a similar 
subject, topic, category, genre, etc. For example, if the song and cUp are associated, the 
recommender may recoirmend, upon the selection of the clip, the associated song and other 
songs which are similar to the associated song. 



ratings provided by the rating means 250. The meta-data may be interpreted in different ways 
for rating them, e.g. as an explicit or implicit feedback firom the user. For example, the media 
contents may be considered as e^licitly associated by the user if the user has explicitly 
initiated reproduction, or the other way of a user's consumption, of both media contents, e.g. 
user's requests for rendering the contents were inputted to the system. The rating means 250 
may highly rate such association of the contents, or a high reliability may be assigned to such 



any language, e.g. XML, and include any amoimt of information about the associated 



Given meta-data storing information about two associated contents, the 



The selection means may be further arranged to function as a recommender of 



The recommender may be arranged to recommend the content on the basis of 
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an association. As shown in Table 1, the (N+1) meta-data record indicates that the user 
initiated displaying the video content C and playing the audio content D. Therefore, the 
contents C and D are associated with the maximum rating, for example with one point as a 
maximal possible rating. If one of the contents was not explicitly chosen for rendering or 
using it otherwise by the user, then the association of these contents may be rated to be lower 
or given less reUabihty. For example, in Table 1 the contents A and B are associated with a 
rating lower than the rating of contents C and D, namely with 0.5 point out of 1, because the 
content B, a song, was played on tibie radio, and the user did not initiate this. 



rating means. For example, the association of the media contents may also be influenced by 
the user's behavior during simultaneous rendering of the contmt, a user's reaction for 
initiating simultaneous rendering of the contents when it was started by the system itself, and 
so on. Other parameters and characteristics of the associated contents, user's usage of the 
associated contents, etc. may be stored in the meta-<lata and used for their rating. 



depending on the number of times of using the associated contents simultaneously. For 
instance, if the contents were often used together, they are given a high rating. 



content and with a third content, the rating means may be arranged to rate said associations of 
the first content with the second content, and the first content with the third content 
differently. For example, the association of the first content and second content may be rated 
to be higher if the first content was more often used with the second content than with tiie 
third content. In another example, the association of the first content may be rated to be 
higher with the second content if the simultaneous usage of the first and second contents is 
more recent than the usage of the first and third contents. Thus, such associated contents are 
rated to be higher if they were recently used. 



"audio 1" 310 is associated with content "video 1" 320. Such an association illustrates tiie 
idea of the present invention about association of the contents of different types. At the same 
time, content "video 1" 320 may be fiirther associated with the other contents of different 
type, e.g. "audio 2" 330, "audio 3" 340, etc. The type of content "music playhsf 350, which 
is also associated with content •'video 1" 320, maybe considered as a separate type of the 
content corresponding to the group of contents of type "music". In turn, contents "audio 1" 
310 and "audio 3" 340 may be associated with content "image 1" 360 and content "video 1" 



Other criteria for evaluating the association of the contents may be used by the 



In a fiirther embodiment, the rating means may rate the associated contents 



In a fijrther embodiment, if the first content is associated with the second 



An example of associated contents is shown with reference to Fig. 3. Content 
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320. In tiris example, if the user selects content "audio 3" and requests the recommender to 
recommend the media content of type different &om type "audio", the recommender may 
recommend contents "image 1" 360 and *Video T' 320 because they are associated with the 
user-selected content "audio 3" 340. 

5 With reference to Fig. 4, an example of media contents used at different time 

in the system is shown. *TV[edia content 1" 410 was used in the system from time TU (411) 
till time T12 (412), "media content 2" 420 - from time T21 (421), "media content 3" 430 - 
from time T31 (431). Contents 410, 420 and 430 have different time lengths for their usage. 
The usage of content 410 was started earlier than the usage of content 420, though this 

10 difference in time (T21-T1 1) is much less than the duration (412-421) of the simultaneous 
usage of contents 410 and 420. The identifying means 120 may be arranged to provide a 
starting time, ending time, time of simultaneous usage of the contents to the associating 
means 210 for including said time information in the corresponding meta-data. The 
associating means may be arranged to estimate a ratio between the duration of simultaneous 

15 usage (412-421) of the corresponding contents and the time difference (T21-T1 1) or 
difference of times when the use of corresponding contents ends. In one example, the 
associating means may be arranged to ignore the simultaneous usage of the contents and the 
respective meta-data may not be established if the duration of simultaneous usage of them is 
much smaller than their usage non-overlapping in time. For example, the duration (412-431) 

20 of simultaneous usage of contents 410 and 430 is rather small with respect to the duration of 
their usage non-overls^ping. The associating means may include a value of said estimated 
ratio in the respective meta-data. In another example, the rating means may rate the 
association of contents based on said ratio. For instance, the association of contents 410 and 
420 may be rated to be higiher than the association of contents 410 and 430 in view of the 

25 considerations above. 

In one of the embodiments of the present invention, the selection means 230 
may enable the user to modify the meta-data in the system. Of coxirse, an appropriate user 
interface including input means, e.g. a keyboard, a mouse, a touch-screen, speech recognition 
means, etc. may be provided, and the output means 240 may be arranged to present the meta- 

30 data to the user via, for instance, a disfplay device, audio reproduction means etc. 

The identifying means, associating means, selection means and rating means 
may be implemented by using a microprocessor (not shown) coupled to a random access 
memory and a read-only memory (ROM) storing a program which, when executed by said 
microprocessor, can perform functions of said means as described above. The storage means 
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220 may be realized with said ROM. The media content may be stored locally in the system 
or downloaded from an external source by means of suitable arranged data receiving means 
(not shown). The realization of such a nriaroprocessor system with corresponding intemal 
and extemal circuits will be apparent to the person skilled in the art and need not be 
5 discussed herein. 

With reference to Fig. 5, the method of the present invention describing the 
operation of the system of the present invention is shown. According to the method of 
operating with different types of media content, the concurrent usage of the first content and 
second contents of different types is identified in step 510, wherein the relationship of said 

10 contents is not known in the system. In step 520, said first and second contents are associated 
as being related by their simultaneous user's usage. For instance, the user may watch a slide 
show of pictures on the display screen together with a certain song being played by the CD 
player in the system, or the user may often listen to some songs when the user would like to 
download certain pictures. Such contents being used by the usct simultaneously are 

15 associated. Embodiments of the method, which correspond to the above-described 
embodiments of the system of the present invention, may be envisaged. 

The various program products may implement the fimctions of the system and 
method of the present invention and may be combined in several ways with the hardware or 
located in different devices. Variations and modifications of the described embodiment are 

20 possible within the scope of the inventive concept. Thus, for example, the use of the verb 'to 
comprise' and its conjugations does not exclude the presence of elements or steps other than 
those defined in a claim. The invention can be implemented by means of hardware 
comprising several distinct elements, and by means of a suitably programmed computer. In 
the device claim enumerating several means, several of these means can be embodied by one 

25 and the same item of hardware. 

A 'computer program' is to be understood to mean any software product 
stored on a computer-readable medium, such as a floppy-disk, downloadable via a network, 
such as the Intemet, or marketable in any other manner. 
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CLAIMS: 



1 . A system for operating with different types of media content, the system being 
arranged to enable a user to use a first content of a first type, characterized in that the system 
comprises: 

identifying means for identifying that the user concurrently uses a second 
5 content of a second type, said second content being unrelated with the first content, and 

associating means for associating said second content with the first content. 

2. The system of claim 1, further comprising storage means arranged to store 
meta-data comprising information pertaining to said associated first and second content. 

10 

3. The system of claim 1, further comprising selection means arranged to select 
the content. 

4. The si^tem of claims 2 and 3, wherein said selection means are further 

15 arranged to identify the first content upon selection of the associated second content and/or to 
identify the second content upon selection of the associated first content, using said 
information stored in the meta-data. 

5. The system of claim 4, wherein said selection means are further arranged to 
20 function as a recommender for recommending the associated first or second content upon a 

user-operable selection of one of said associated second and first content, respectively, using 
said selection means. 

6. The system of claim 4, further comprising output means arranged to 
25 simultaneously output said associated first and second content. 

7. The system of claim 2 and 3, wherein said selection means are further 
arranged to user-operably modify said meta-data. 
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8, The system of claim 1, wherein said identifying means is arranged to identify 

a user's usage of a third content of a second or other type, said usage being concurrent to said 
user's usage of the first content, and said third content being lonrelated with the first content, 
and wherein said associating means is arranged to associate said third and first content, the 
5 system further comprising rating means arranged to rate said association of the first content 
with the second content and/or with the third content 



9. The system of claim 1 , comprising a plurality of devices, each device 
including output means arranged to output at least one type of the media content, and/or input 

10 means arranged to obtain at least one type of the media content. 

10. The system of any one of the preceding claims, wherein said first and second 
content correspond to video and audio content. 

15 11. A method of operating with different types of media content, the method 

comprising a step of identifying a user's usage of a first content of a first type, characterized 
in that the method further comprises 

a step of identifying that the user concurrently uses a second content of a 
second type, said second content being unrelated with the first content, and 

20 - a step of associating said second content with the first content. 



12. A computer program product enabling a programmable device, when 

executing said computer program product, to function as the system as defined in claim 1 . 
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